Dynamically Resolving Incidents via Event Rules Email Integrations

#1

I am having trouble understanding how to automatically resolve events that are being routed to an Event Rules integration via Email.

On an individual service, the Email integration allows you to “Link incoming emails to the same alert” and define rules that create and resolve the incidents based on the payload of the Email, IE, subject has “Problem Created” - create incident, subject has “Problem Close” - close incident.

Unfortunately, I am not seeing the ability to configure similar behavior via Event Rules + Email.

The use case is this - we want to configure the source sending Email notifications to send to a single Event Rules Integration. From there, we would like certain messages to route to a “Business Hours” service, and other messages to route to a “24x7” service. We would like these to automatically close, as well, when the appropriate email is sent.

That incident creation is easy to setup, but what is not easily achieved is the ability to Resolve those same Incidents when the associated “Problem Closed” message is sent. Based on my testing, it will just create another incident with “Problem Closed” message, result being 2 incidents for one event - Problem Open and Problem Closed incident(s).

Is there someone with experience in setting something up similar to this that may have some insight on best practices?

#2

By default, the email subject line is your deduplication key so if those are different, you won’t be able to match up the “Problem Close” with the “Problem Created” emails.

In most cases, you’ll need to create a unique trigger rule for the email that has a subject line of “Problem Created…” which extracts/creates a unique deduplication key and triggers the incident, and another unique resolution rule for the email that has a subject line of “Problem Close…” that extracts/creates the exact same deduplication key and resolves the incident.

Give this a shot!

#3

Good day!

To be clear - I have good success with using the Service Email integration to both Open and Resolve incidents.

The problem I am having is that I am not able to Resolve via Event Rules Email integration, that routes to those same 2 services.

Here are more specific details around these Services.

Service name: ITSM - Test Business Hours Service
email-itsm-businesshours-integration@privacystar.pagerduty.com

Subject to open:
Alert Open : A warning foo-bar has occurred : Warning
Subject to close:
Alert Closed : A warning foo-bar has occurred : Warning

Service name: ITSM - Test 24x7 Service

email-itsm-24x7-integration@privacystar.pagerduty.com

Subject to open:
Alert Open : A critical foo-bar has occurred : Critical

Subject to close:
Alert Closed : A critical foo-bar has occurred : Critical

These rules work very well on the Service Email integration settings:

Open and resolve alerts based on custom rules:

Trigger an alert if any of the following conditions apply
The email subject contains Alert Open
Deduplicate based on the alert key found between " : " and the very end of the email subject
Resolve an alert if any of the following conditions apply
The email subject contains Alert Closed
Deduplicate based on the alert key found between " : " and the very end of the email subject
And create a generic alert for any email that does not match any of the above rules

BUT, when I use an Event Rule to route the same exact Email Subjects to the underlying Services, the “Alert Closed” Emails do not resolve the incident as they do when hitting the Service directly.

I do not see configuration options on the Event Rules to create similar behavior when interacting with it, as opposed to directly interacting with the underlying Service.

#4

The key to success is in the deduplication values matching for your trigger and resolve event rules. Are you extracting a value from the Alert Open and Alert Closed emails that will always match?

#5

I’ve confirmed from working with support the following.

“If the email alert is sent to the Rulesets email address, the settings at the Service integration level will not apply to the Alert.”

There is a mismatch in the capabilities of a Service level Email integration and a Event Rules Email Integration Ruleset.

The ability to perform deduplication values for trigger/resolve does not appear to exist in Event Rules.